Identifying Proto-Languages using Word Embeddings

1. Overview

How it started

In a previous article (here), I discussed how exploring multi-lingual text data on qdrant's UI lead to the following discoveries (1) a heuristic that uses the count of characters associated with a specific language achieved performance comparable (if not better) to nomic-embed-text-v2-moe:latest and gemini-embedding-001 in identifying the language in the text - both LLM’s support multiple languages (2) nomic-embed-text:latest which only supports English language achieved comparable performance to gemini-embedding-001 in the language identification task. The languages used in the task can be further grouped into their respective proto-languages - the so called primordial language from which a set of languages evolved. For instance, Amharic, Arabic, Hausa, and Somali are postulated to share a common ancestor (mother lanugage) termed Afro-Asiatic, while Tamil, Spanish and English belong to the Dravidian language group. This led to the following questions:

Could word representations be used in proto-language identification?
Which of the LLMs is best suited to identfying the proto languages of the text?

Key Findings - Good for the gander: the embedding models identified the proto-language (language group) of the text roughly as well as they identified the language itself (see here). - Broad language support did not help: nomic-embed-text-v2-moe:latest, which supports almost every language in the study, had the lowest mutual information and recall. - Small language groups are the hardest: performance was weakest for language groups made up of one or two languages, e.g. Turkic and Kra-Dai (Thai).

So What - Language identification could potentially be split into a two-step process consisting of first identifying the proto-language (language group) and then identifying the specific language. - gemini-embedding-001 is also better at retrieving translations of the same verse across languages (here), so a single model could serve both semantic search and language-group identification.

Code

import warnings
from numba.core.errors import NumbaWarning
warnings.filterwarnings("ignore", category=NumbaWarning)

from vectoriser import get_qdrant
from query_qdrant import (load_sample, load_sample_where, 
                    get_payload_keys, get_payload, 
                    sample_records, extract_payload, 
                    fetch_records_for_dataframe)

from sklearn.cluster import KMeans
from sklearn.neighbors import KNeighborsClassifier, NearestNeighbors
from sklearn.preprocessing import LabelEncoder
from sklearn.decomposition import PCA
from sklearn.metrics import normalized_mutual_info_score, recall_score, precision_score
from sklearn.pipeline import Pipeline

import numpy as np
from itertools import combinations
import pandas as pd
import umap
from matplotlib import pyplot as plt
import seaborn as sb
from umap import UMAP
import umap.plot
import matplotlib.ticker as ticker
import regex
from lang_util import detect_language, language_map, language_family_map
from pandas.io.formats.style import Styler

1.2 Approach

To address the two questions (above), I adopted the following approach:

Word embeddings and their corresponding text and attributes were sampled from three collections on qdrant where each collection corresponds to vectors produced using either gemini-embedding-001, nomic-embed-text-latest, or nomic-embed-text-v2-moe:latest. The text consists of quranic script in a variety of languages obtained from tanzil.net. Details of the text data and the LLMs is provided in Section 2.0.
To enable like-for-like comparison between the vectors, samples were first collected from one collection and the text from the collection was used to obtain a matching sample for each of the other collections i.e. the only thing that differs between the three collections is their word vectors (word embeddings).
For each sample from the collections, a UMAP model was fitted on a subset of the sample i.e. the training set, obtaining a projection of the data on a 2-dimensional Riemannian manifold. PCA was first used to reduce the dimensions of the vectors before fitting the UMAP model. Subsequently a knn-classifier is fitted on the projected data, assigning a label (predicted language group) to each record. This is similar to the BERTopic approach.

flowchart TB

A[word embeddings] -->|sampling| B[sample]
B -->|UMAP - Dimensionality Reduction| C[Projections]
C --> |KNN Classification| E[Language Classification]
E-->|Metrics Calculation|F[Comparison]

2.0 Data

The vectorised quranic text obtained from tanzil.net is stored in the following collections on a qdrant instance running locally on a homelab cluster. Details of each collection and the vectorisers used is provided below:

Collection (word embeddings)	Number of Records	Vector Dimensions	Vectoriser	Source	Vectoriser No. Parameters (Millions)	Model Type
`quran-embedding-prefect-gemini-embedding-001`	249,442	768	`gemini-embedding-001`	Google	Undisclosed	Transformer (Gemini-initialised, decoder-based)
`quran-embedding-prefect-nomic-embed-text-latest`	249,442	768	`nomic-embed-text:latest`	Ollama (Local Instance)	137	BERT
`quran-embedding-prefect-nomic-embed-text-v2-moe-latest`	249,442	768	`nomic-embed-text-v2-moe:latest`	Ollama (Local Instance)	475.29	BERT - MoE

A list of the languages covered in this study and whether or not they are supported by either of the embedding models is provided below

Code	Language	gemini-embedding-001	nomic v1.5 (nomic-embed-text-latest)	nomic-embed-text-v2-moe
am	Amharic	✓	✗	✓
ar	Arabic	✓	✗	✓
bs	Bosnian	✓	✗	✓
dv	Divehi	✓	✗	✗
en	English	✓	✓	✓
es	Spanish	✓	✗	✓
fa	Persian	✓	✗	✓
ha	Hausa	✓	✗	✓
nl	Dutch	✓	✗	✓
ru	Russian	✓	✗	✓
sd	Sindhi	✓	✗	✗
so	Somali	✓	✗	✓
sq	Albanian	✓	✗	✓
ta	Tamil	✓	✗	✓
tg	Tajik	✓	✗	✓
th	Thai	✓	✗	✓
tr	Turkish	✓	✗	✓
ur	Urdu	✓	✗	✓
uz	Uzbek	✓	✗	✓
zh	Chinese	✓	✗	✓

2.1 Sampling

Samples were collected from the collections starting with one collection first and then using the text in the first collection to sample records matching the same text from the other two collections.
The data is largely comprised of text in English (en) therefore some downsampling was applied to prevent skewing the results.

Code

def downsample_majority(df,samp_col="language",random_state=1235813):
    props = df.groupby([samp_col]).size().to_dict()
    sorted_props = sorted(props, key=lambda e:props[e],reverse=True)
    maj = sorted_props[0]
    avg_minority = int(np.ceil(np.mean([props[e] for e in props if e!=maj])))
    A = df[df[samp_col].eq(maj)].sample(avg_minority, random_state=random_state)
    B = df[~df[samp_col].eq(maj)]
    C = pd.concat([A,B]).reset_index(drop=True)

    return C

Code

collections = {"gemini-embedding-001":'quran-embedding-prefect-gemini-embedding-001',
               "nomic-embed-text:latest":'quran-embedding-prefect-nomic-embed-text-latest', 
                "nomic-embed-text-v2-moe:latest":'quran-embedding-prefect-nomic-embed-text-v2-moe-latest'}

Code

sample_prop = 0.4

Code

init = "gemini-embedding-001"
vectors_main, main_df = sample_records(collections[init],frac=sample_prop,exclude=[])
main_df['language_description'] = main_df['language'].map(language_map)
main_df['language_group'] = main_df['language'].map(language_family_map)

Code

freq = main_df.groupby(['language_group','language','language_description'],dropna=False).size()
freq = pd.DataFrame(freq).rename(columns={0:"Number of Occurrences"})
freq

			Number of Occurrences
language_group	language	language_description
Afroasiatic	am	Amharic	2490
	ar	Arabic	4920
	ha	Hausa	2457
	so	Somali	2547
Dravidian	ta	Tamil	2519
Indo-European	bs	Bosnian	2472
	dv	Divehi	2493
	en	English	42337
	es	Spanish	2538
	fa	Persian	2458
	nl	Dutch	2489
	ru	Russian	2478
	sd	Sindhi	2521
	sq	Albanian	2527
	tg	Tajik	2517
	ur	Urdu	4909
Kra-Dai	th	Thai	2592
Sino-Tibetan	zh	Chinese	5035
Turkic	tr	Turkish	2469
Turkic	uz	Uzbek	2482
NaN		NaN	2527

Code

samples = {}
samples[init] = downsample_majority(main_df)

Code

vectors = {}
row_pos = pd.Series(np.arange(len(main_df)), index=main_df['id'])
vectors[init] = vectors_main[row_pos.loc[samples[init]['id']].to_numpy(), :]

Code

samples[init].groupby(['language']).size()
%store init

Stored 'init' (str)

An example of attributes of vectors sampled from the collection corresponding to gemini-embedding-001 is shown below. The column text in the table is what the vectors were generated from.

Code

samples[init].head()

	index	id	language	name	chapter	verse	text	surah_name	revelation_order	revelation_type	revelation_note	language_description	language_group
0	236019	f2627bd6-f23a-54e5-a1a6-5bbdbad95756	en	ahmedraza	88	7	Which neither give strength nor satisfy the hu...	Al-Ghaashiya	68	Meccan	NaN	English	Indo-European
1	248871	ff6d1d6b-bf5d-52c7-962a-966cc590144a	en	qarai	68	26	But when they saw it, they said, ‘We have inde...	Al-Qalam	2	Medinan	Except 17-33 and 48-50, from Medina	English	Indo-European
2	217314	df0c4ee7-8643-57f8-86ad-e5c489867bed	en	sahih	10	77	Moses said, "Do you say [thus] about the truth...	Yunus	51	Meccan	Except 40, 94, 95, 96, from Medina	English	Indo-European
3	4029	042bb1ea-609d-5059-bacc-dd2fe3110639	en	qaribullah	37	63	We have made this (tree) a trial for the evild...	As-Saaffaat	56	Meccan	NaN	English	Indo-European
4	24268	18e67fb4-1203-54f0-bf46-d011a8464c28	en	sahih	6	95	Indeed, Allah is the cleaver of grain and date...	Al-An'aam	55	Meccan	Except 20, 23, 91, 93, 114, 151, 152, 153, fro...	English	Indo-European

A sample equivalent to that collected from quran-embedding-prefect-gemini-embedding-001 was collected from the other two collections detailed in Section 2.0

Code

filter_columns = ['language','name','chapter','verse','text']

colls  = [coll for coll in collections if coll !=init]
for c in colls:
    if collections[c]!= None:
        print("Processing {}".format(c))
        matching_records = fetch_records_for_dataframe(
            samples[init],
            collections[c],
            filter_cols = filter_columns,
            batch_size = 200,
            with_vectors = True
        )
        samp, vec = extract_payload(matching_records, exclude=[])
        samples[c] = pd.DataFrame(samp) 
        vectors[c] = vec

        assert len(samples[c]) == len(samples[init]), (
            f"{c}: expected {len(samples[init])} matching records, got {len(samples[c])} "
            "- some rows had no exact match, so the like-for-like comparison "
            "across collections no longer holds."
        )
    else:
        samples[c] = None
        vectors[c] = None

Processing nomic-embed-text:latest
Processing nomic-embed-text-v2-moe:latest

An example of attributes of sample records stored in quran-embedding-prefect-nomic-embed-text-latest is shown below

Code

samples['nomic-embed-text:latest'].head()

	id	language	name	chapter	verse	text	surah_name	revelation_order	revelation_type	revelation_note
0	f2627bd6-f23a-54e5-a1a6-5bbdbad95756	en	ahmedraza	88	7	Which neither give strength nor satisfy the hu...	Al-Ghaashiya	68	Meccan	NaN
1	ff6d1d6b-bf5d-52c7-962a-966cc590144a	en	qarai	68	26	But when they saw it, they said, ‘We have inde...	Al-Qalam	2	Medinan	Except 17-33 and 48-50, from Medina
2	df0c4ee7-8643-57f8-86ad-e5c489867bed	en	sahih	10	77	Moses said, "Do you say [thus] about the truth...	Yunus	51	Meccan	Except 40, 94, 95, 96, from Medina
3	042bb1ea-609d-5059-bacc-dd2fe3110639	en	qaribullah	37	63	We have made this (tree) a trial for the evild...	As-Saaffaat	56	Meccan	NaN
4	18e67fb4-1203-54f0-bf46-d011a8464c28	en	sahih	6	95	Indeed, Allah is the cleaver of grain and date...	Al-An'aam	55	Meccan	Except 20, 23, 91, 93, 114, 151, 152, 153, fro...

A comparison of the statistical signature of the sample from the collection associated with gemini-embedding-001 to collections associated with the other models is shown below. The figure shows that the samples are equivalent i.e. the text, chapter, and verse match across the three samples.

Code

fig, ax = plt.subplots(3,3,figsize=(20,20), sharex=False)
ax = ax.flatten()

for x,c in zip([0, 3, 6],collections.keys()):
    samples[c].groupby('language').size().plot.bar(ax=ax[x])
    ax[x].set_title(c)

for x,c in zip([1,4,7],collections.keys()):
    samples[c].groupby('chapter').size().plot.bar(ax=ax[x])
    ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6))
    ax[x].set_title(c)

for x,c in zip([2,5,8],collections.keys()):
    samples[init].groupby('verse').size().plot.bar(ax=ax[x])
    ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6))
    ax[x].set_title(init)

for a in ax: 
    a.set_ylabel("Number of Occurrences")

Code

for col in samples:
    if collections[col]!=None:
        samples[col]['language_description'] = samples[col]['language'].map(language_map)
        samples[col]['language_group'] = samples[col]['language'].map(language_family_map)

Code

counts = {}
for c in samples:
    if collections[c] != None:
        count = samples[c].fillna('n/a').groupby(['language_group','language','language_description'],dropna=False).size()
        counts[c] = count

A comparison of the number of records in each sample is provided below. The sample is largely dominated by english scripts. Records with no assigned language (labelled “n/a”) are in the uthmaniya script which is Arabic.

Code

pd.DataFrame(counts.values(),index = counts.keys()).T

			gemini-embedding-001	nomic-embed-text:latest	nomic-embed-text-v2-moe:latest
language_group	language	language_description
Afroasiatic	am	Amharic	2490	2490	2490
	ar	Arabic	4920	4920	4920
	ha	Hausa	2457	2457	2457
	so	Somali	2547	2547	2547
Dravidian	ta	Tamil	2519	2519	2519
Indo-European	bs	Bosnian	2472	2472	2472
	dv	Divehi	2493	2493	2493
	en	English	2872	2872	2872
	es	Spanish	2538	2538	2538
	fa	Persian	2458	2458	2458
	nl	Dutch	2489	2489	2489
	ru	Russian	2478	2478	2478
	sd	Sindhi	2521	2521	2521
	sq	Albanian	2527	2527	2527
	tg	Tajik	2517	2517	2517
	ur	Urdu	4909	4909	4909
Kra-Dai	th	Thai	2592	2592	2592
Sino-Tibetan	zh	Chinese	5035	5035	5035
Turkic	tr	Turkish	2469	2469	2469
Turkic	uz	Uzbek	2482	2482	2482
n/a		n/a	2527	2527	2527

3.0 Language Identification Using Word Representations

3.1 PCA + UMAP Parameters

The 768-dimensional vectors are first reduced with PCA before UMAP is fitted on the result - this cuts UMAP’s nearest-neighbor search cost substantially, since that cost scales with the input dimensionality. Details of the parameters used for the PCA step and the subsequent UMAP fit are provided below

Code

pca_params = dict(n_components=50, random_state=1235813)

Code

umap_params = dict(n_neighbors=30,
            verbose=True,
            n_epochs=300,
            # a fixed seed makes the projection reproducible; UMAP then runs
            # single-threaded, so n_jobs is not set
            random_state=1235813)

3.2 KNN Classifier Parameters

The parameters used for fitting the KNeighborsClassifier is provided below

Code

knn_params = dict(
    n_jobs=-1,
    n_neighbors = 20, 
    weights = 'uniform'
)

3.3 Modelling

Code

def train_test_split(df, id_col="id", train_frac=0.7):
    """ split a dataframe into train and test based on id_col """
    tr_id = df.sample(frac=train_frac, random_state=np.random.RandomState(123456))[id_col]
    tst_id = df[~df[id_col].isin(tr_id)][id_col]

    tr_flag = df.id.isin(tr_id)
    tst_flag = df.id.isin(tst_id)

    return tr_flag, tst_flag

def train_umap(vec, pca_kwargs=None, **kwargs):
    """ reduce dimensionality with PCA, then fit a umap model on the reduced vectors """
    dim_model = Pipeline([
        ('pca', PCA(**(pca_kwargs or {}))),
        ('umap', UMAP(**kwargs)),
    ])
    dim_red = dim_model.fit_transform(vec)

    return dim_red, dim_model

def fit_knn_classifier(red_vec,tr_tar,**kwargs):
    """ fit a knn classifier """ 
    k_class = KNeighborsClassifier(**kwargs)

    k_class.fit(red_vec, tr_tar)

    return k_class

def project(dim_model, vec, train_vec):
    """ project vec with the fitted PCA+UMAP model (dim_model), returning the
    projection and the number of points that needed a fallback.

    UMAP's transform embeds a point as NaN when its approximate nearest-neighbour
    search finds no neighbours in the training data. Such points are placed at
    the embedding of their exact nearest training point in PCA space instead.
    """
    proj = dim_model.transform(vec)
    missing = np.isnan(proj).any(axis=1)

    if missing.any():
        pca = dim_model.named_steps['pca']
        nn = NearestNeighbors(n_neighbors=1).fit(pca.transform(train_vec))
        nearest = nn.kneighbors(pca.transform(vec[missing]), return_distance=False)[:, 0]
        proj[missing] = dim_model.named_steps['umap'].embedding_[nearest]

    return proj, int(missing.sum())

Code

def compute_norm_mutual_info(df,tar_col,pred_col):
    """ Computes the mutual information based on columns tar_col (target) and pred_col (prediction) """

    tar = df[tar_col].to_numpy()
    pred = df[pred_col].to_numpy()

    mi = normalized_mutual_info_score(tar,pred)

    return mi

Code

def compute_recall(df,tar_col, pred_col,average="macro"):
    """Calculates the recall based on tar_col (target) and pred_col (prediction) """

    tar = df[tar_col].to_numpy()
    pred = df[pred_col].to_numpy()

    rec = recall_score(tar, pred,average=average,zero_division=np.nan)

    return rec

Code

def compute_precision(df, tar_col,pred_col,average="macro"):
    """ Calculates the precision based on tar_col (target) and pred_col (prediction) """

    tar = df[tar_col].to_numpy()
    pred = df[pred_col].to_numpy()

    prec = precision_score(tar,pred, average=average,zero_division=np.nan)

    return prec

Code

def style_dataframe_heatmap(
    df: pd.DataFrame, cmap: str = "YlGnBu", precision: int = 2
        ) -> Styler:
    """Returns a pandas Styler with a background heatmap pinned strictly between 0 and 1."""
    return (
        df.style.background_gradient(cmap=cmap, vmin=0.0, vmax=1.0)
        .format(f"{{:.{precision}f}}")
        .set_properties(**{"text-align": "center"})
    )

Code

def style_summary_table(
    df: pd.DataFrame,
    cmap: str = "Blues",
    vmin: float = 0.0,
    vmax: float = 1.0,
) -> Styler:
    """Applies a background gradient to all rows except the first row (e.g.

    'Number of Languages').
    """
    # First row is the count row (e.g. 'Number of Languages'); the rest are stats.
    count_row = df.index[0]
    stat_rows = df.index[1:]

    styled = (
        df.style.background_gradient(
            cmap=cmap,
            vmin=vmin,
            vmax=vmax,
            subset=pd.IndexSlice[stat_rows, :],
        )
        .format("{:.0f}", subset=pd.IndexSlice[[count_row], :])
        .format("{:.2f}", subset=pd.IndexSlice[stat_rows, :])
        .set_properties(**{"text-align": "center"})
    )

    return styled

The samples are split into a train and test set for validation purposes and a UMAP model is fitted on the portion of the sample tagged as training

Code

train_flag = {} 
test_flag = {} 
dim_red = {}
umap_model = {}

for c in collections:
    print("Processing {}".format(c))
    train_, test_ = train_test_split(samples[c])
    dim_r, umap_m = train_umap(vectors[c][train_, :],
                                 pca_kwargs=pca_params,
                                 **umap_params)
    train_flag[c] = train_ 
    test_flag[c] = test_
    dim_red[c] = dim_r 
    umap_model[c] = umap_m

Processing gemini-embedding-001
UMAP(n_epochs=300, n_jobs=1, n_neighbors=30, random_state=1235813, verbose=True)
Sun Nov  1 21:11:18 2026 Construct fuzzy simplicial set
Sun Nov  1 21:11:18 2026 Finding Nearest Neighbors
Sun Nov  1 21:11:18 2026 Building RP forest with 15 trees
Sun Nov  1 21:11:24 2026 NN descent for 15 iterations
     1  /  15
     2  /  15
     3  /  15
    Stopping threshold met -- exiting after 3 iterations
Sun Nov  1 21:11:38 2026 Finished Nearest Neighbor Search
Sun Nov  1 21:11:44 2026 Construct embedding
    completed  0  /  300 epochs
    completed  30  /  300 epochs
    completed  60  /  300 epochs
    completed  90  /  300 epochs
    completed  120  /  300 epochs
    completed  150  /  300 epochs
    completed  180  /  300 epochs
    completed  210  /  300 epochs
    completed  240  /  300 epochs
    completed  270  /  300 epochs
Sun Nov  1 21:12:40 2026 Finished embedding
Processing nomic-embed-text:latest
UMAP(n_epochs=300, n_jobs=1, n_neighbors=30, random_state=1235813, verbose=True)
Sun Nov  1 21:12:43 2026 Construct fuzzy simplicial set
Sun Nov  1 21:12:43 2026 Finding Nearest Neighbors
Sun Nov  1 21:12:43 2026 Building RP forest with 15 trees
Sun Nov  1 21:12:43 2026 NN descent for 15 iterations
     1  /  15
     2  /  15
    Stopping threshold met -- exiting after 2 iterations
Sun Nov  1 21:12:48 2026 Finished Nearest Neighbor Search
Sun Nov  1 21:12:48 2026 Construct embedding
    completed  0  /  300 epochs
    completed  30  /  300 epochs
    completed  60  /  300 epochs
    completed  90  /  300 epochs
    completed  120  /  300 epochs
    completed  150  /  300 epochs
    completed  180  /  300 epochs
    completed  210  /  300 epochs
    completed  240  /  300 epochs
    completed  270  /  300 epochs
Sun Nov  1 21:20:45 2026 Finished embedding
Processing nomic-embed-text-v2-moe:latest
UMAP(n_epochs=300, n_jobs=1, n_neighbors=30, random_state=1235813, verbose=True)
Sun Nov  1 21:20:47 2026 Construct fuzzy simplicial set
Sun Nov  1 21:20:48 2026 Finding Nearest Neighbors
Sun Nov  1 21:20:48 2026 Building RP forest with 15 trees
Sun Nov  1 21:20:48 2026 NN descent for 15 iterations
     1  /  15
     2  /  15
     3  /  15
     4  /  15
    Stopping threshold met -- exiting after 4 iterations
Sun Nov  1 21:20:54 2026 Finished Nearest Neighbor Search
Sun Nov  1 21:20:54 2026 Construct embedding
    completed  0  /  300 epochs
    completed  30  /  300 epochs
    completed  60  /  300 epochs
    completed  90  /  300 epochs
    completed  120  /  300 epochs
    completed  150  /  300 epochs
    completed  180  /  300 epochs
    completed  210  /  300 epochs
    completed  240  /  300 epochs
    completed  270  /  300 epochs
Sun Nov  1 21:22:05 2026 Finished embedding

/home/jovyan/work/kadamu/taleb/lib/python3.13/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(
/home/jovyan/work/kadamu/taleb/lib/python3.13/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(
/home/jovyan/work/kadamu/taleb/lib/python3.13/site-packages/umap/umap_.py:1952: UserWarning: n_jobs value 1 overridden to 1 by setting random_state. Use no seed for parallelism.
  warn(

Visual inspection of the projections shows distinct clusters, each comprised primarily of a single language group, for all three vectorisers. The colours correspond to the actual language group of the text.

Code

fig, ax = plt.subplots(2,2,figsize=(15,15))
ax = ax.flatten()

for i,c in enumerate(collections):
    labels = samples[c][train_flag[c]].language_group.fillna("n/a")
    umap.plot.points(umap_model[c].named_steps['umap'],labels=labels,ax=ax[i])
    ax[i].set_title(c)
    ax[i].set_xlabel("Dimension - 1")
    ax[i].set_ylabel("Dimension -2")

    leg = ax[i].get_legend()
    if leg != None:
        leg.set_bbox_to_anchor((1.02, 1))
        leg.set_loc("upper left")


plt.tight_layout()

Code

labels

0        Indo-European
2        Indo-European
3        Indo-European
4        Indo-European
5        Indo-European
             ...      
60306    Indo-European
60307    Indo-European
60309    Indo-European
60310    Indo-European
60311    Indo-European
Name: language_group, Length: 42218, dtype: str

Code

tar_encoder = {}
for c in collections: 
    tar_encoder[c] = LabelEncoder()
    language_values = samples[c].language_group.fillna("n/a")
    # detect_language (used later for heur_lang) can return '' when no candidate
    # clears MIN_CONFIDENCE, even though no ground-truth row is ever ''. Include
    # it as a known class up front so transform() never hits an unseen label.
    tar_encoder[c].fit(pd.concat([language_values, pd.Series([''])], ignore_index=True))
    target = tar_encoder[c].transform(language_values)
    samples[c]['target_idx'] = pd.Series(target)

Code

knn_clf = {}
for c in collections:
    knn_clf[c] = fit_knn_classifier(dim_red[c],samples[c].loc[train_flag[c],'target_idx'],**knn_params)

Code

train_pred = {}
test_pred = {}
for c in collections:
    tr_f = train_flag[c]
    te_f = test_flag[c]
    train_vec = vectors[c][tr_f, :]

    tr_proj, tr_missing = project(umap_model[c], train_vec, train_vec)
    te_proj, te_missing = project(umap_model[c], vectors[c][te_f, :], train_vec)
    print(f"{c}: {tr_missing} train and {te_missing} test points had no UMAP "
          "neighbours and were placed at their nearest training point")

    train_pred[c] = knn_clf[c].predict(tr_proj)
    test_pred[c] = knn_clf[c].predict(te_proj)

    samples[c]['knn_predictions'] = np.nan
    samples[c].loc[tr_f, 'knn_predictions'] = train_pred[c]
    samples[c].loc[te_f, 'knn_predictions'] = test_pred[c]
    samples[c]['knn_predictions'] = samples[c]['knn_predictions'].astype(int)

    samples[c].loc[tr_f,'Train/Test'] = "Train"
    samples[c].loc[te_f, 'Train/Test'] = "Test"

Sun Nov  1 21:22:10 2026 Building hub-based search tree
Sun Nov  1 21:22:17 2026 Forward diversification reduced edges from 1266540 to 421183
Sun Nov  1 21:22:19 2026 Reverse diversification reduced edges from 421183 to 421183
Sun Nov  1 21:22:21 2026 Degree pruning reduced edges from 483844 to 483844
Sun Nov  1 21:22:21 2026 Resorting data and graph based on tree order
Sun Nov  1 21:22:22 2026 Building and compiling search function
    completed  0  /  100 epochs
    completed  10  /  100 epochs
    completed  20  /  100 epochs
    completed  30  /  100 epochs
    completed  40  /  100 epochs
    completed  50  /  100 epochs
    completed  60  /  100 epochs
    completed  70  /  100 epochs
    completed  80  /  100 epochs
    completed  90  /  100 epochs
gemini-embedding-001: 0 train and 0 test points had no UMAP neighbours and were placed at their nearest training point
Sun Nov  1 21:22:32 2026 Building hub-based search tree
Sun Nov  1 21:22:32 2026 Forward diversification reduced edges from 1266540 to 424702
Sun Nov  1 21:22:32 2026 Reverse diversification reduced edges from 424702 to 424702
Sun Nov  1 21:22:32 2026 Degree pruning reduced edges from 527820 to 527784
Sun Nov  1 21:22:32 2026 Resorting data and graph based on tree order
Sun Nov  1 21:22:32 2026 Building and compiling search function
    completed  0  /  100 epochs
    completed  10  /  100 epochs
    completed  20  /  100 epochs
    completed  30  /  100 epochs
    completed  40  /  100 epochs
    completed  50  /  100 epochs
    completed  60  /  100 epochs
    completed  70  /  100 epochs
    completed  80  /  100 epochs
    completed  90  /  100 epochs
nomic-embed-text:latest: 0 train and 120 test points had no UMAP neighbours and were placed at their nearest training point
Sun Nov  1 21:22:43 2026 Building hub-based search tree
Sun Nov  1 21:22:43 2026 Forward diversification reduced edges from 1266540 to 446401
Sun Nov  1 21:22:43 2026 Reverse diversification reduced edges from 446401 to 446401
Sun Nov  1 21:22:43 2026 Degree pruning reduced edges from 538314 to 538052
Sun Nov  1 21:22:43 2026 Resorting data and graph based on tree order
Sun Nov  1 21:22:43 2026 Building and compiling search function
    completed  0  /  100 epochs
    completed  10  /  100 epochs
    completed  20  /  100 epochs
    completed  30  /  100 epochs
    completed  40  /  100 epochs
    completed  50  /  100 epochs
    completed  60  /  100 epochs
    completed  70  /  100 epochs
    completed  80  /  100 epochs
    completed  90  /  100 epochs
nomic-embed-text-v2-moe:latest: 0 train and 0 test points had no UMAP neighbours and were placed at their nearest training point

4.0 Results

4.1 Performance of Word Embeddings in Proto-Language Identification

The performance of the umap+knn-classification approach on all three collections was assessed on the test set using Normalised Mutual Information, Recall and Precision. gemini-embedding-001 performed best on all three metrics, followed closely by nomic-embed-text:latest, with nomic-embed-text-v2-moe:latest trailing on mutual information and recall. The strong performance of nomic-embed-text:latest is surprising, particularly because it was developed primarily for English (en).
In a separate exploration (here), I showed how gemini-embedding-001 performs better at retrieving translations that share the same verse both in the same language and cross-language and therefore is better suited for an application that requires looking beyond the characters in the text.

Code

vectoriser_props = {
    "gemini-embedding-001":{"dimension":768, "n_params":"Undisclosed"}, 
    "nomic-embed-text:latest":{"dimension":768,"n_params":"137 Million"}, 
    "nomic-embed-text-v2-moe:latest":{"dimension":768, "n_params":"475.29 Million"},
}

Code

collections

{'gemini-embedding-001': 'quran-embedding-prefect-gemini-embedding-001',
 'nomic-embed-text:latest': 'quran-embedding-prefect-nomic-embed-text-latest',
 'nomic-embed-text-v2-moe:latest': 'quran-embedding-prefect-nomic-embed-text-v2-moe-latest'}

Code

group_cols = ['Collection','Vectoriser','Vectoriser Dimensions','Vectoriser Parameters','Train/Test']
mi = []
rec = []
prec = []
for c in collections:
    samples[c]['Collection'] = c 
    
    samples[c]['Vectoriser'] = collections[c]
    samples[c]['Vectoriser Dimensions'] = vectoriser_props[c]["dimension"]
    samples[c]['Vectoriser Parameters'] = vectoriser_props[c]["n_params"]

    d = samples[c].groupby(group_cols).apply(lambda df:compute_norm_mutual_info(df, 'target_idx','knn_predictions'))
    mi.append(d)

    d1 = samples[c].groupby(group_cols).apply(lambda df:compute_recall(df, 'target_idx', 'knn_predictions'))
    rec.append(d1)

    d2 = samples[c].groupby(group_cols).apply(lambda df:compute_precision(df, 'target_idx', 'knn_predictions'))
    prec.append(d2)

Normalised Mutual Information - gemini-embedding-001 (0.90) and nomic-embed-text:latest (0.88), which only supports English, produced vectors that are similarly informative about the language group of the text, well ahead of nomic-embed-text-v2-moe:latest (0.72).

Code

mi_res = pd.concat(mi).unstack()
mi_res

			Train/Test	Test	Train
Collection	Vectoriser	Vectoriser Dimensions	Vectoriser Parameters
gemini-embedding-001	quran-embedding-prefect-gemini-embedding-001	768	Undisclosed	0.886526	0.897086
nomic-embed-text-v2-moe:latest	quran-embedding-prefect-nomic-embed-text-v2-moe-latest	768	475.29 Million	0.715863	0.736630
nomic-embed-text:latest	quran-embedding-prefect-nomic-embed-text-latest	768	137 Million	0.889633	0.898025

Recall - gemini-embedding-001 achieved the highest recall (0.95), followed by nomic-embed-text:latest (0.91) and nomic-embed-text-v2-moe:latest (0.79).

Code

rec_res = pd.concat(rec).unstack()
rec_res

			Train/Test	Test	Train
Collection	Vectoriser	Vectoriser Dimensions	Vectoriser Parameters
gemini-embedding-001	quran-embedding-prefect-gemini-embedding-001	768	Undisclosed	0.946168	0.951814
nomic-embed-text-v2-moe:latest	quran-embedding-prefect-nomic-embed-text-v2-moe-latest	768	475.29 Million	0.771297	0.763828
nomic-embed-text:latest	quran-embedding-prefect-nomic-embed-text-latest	768	137 Million	0.919187	0.921566

Precision - gemini-embedding-001 achieved the best precision (0.96), followed by nomic-embed-text-v2-moe:latest (0.92) and nomic-embed-text:latest (0.89). nomic-embed-text-v2-moe:latest is therefore conservative: when it assigns a language group it is usually right, but it misses more records than the other two models.

Code

prec_res = pd.concat(prec).unstack()
prec_res

			Train/Test	Test	Train
Collection	Vectoriser	Vectoriser Dimensions	Vectoriser Parameters
gemini-embedding-001	quran-embedding-prefect-gemini-embedding-001	768	Undisclosed	0.952532	0.959297
nomic-embed-text-v2-moe:latest	quran-embedding-prefect-nomic-embed-text-v2-moe-latest	768	475.29 Million	0.927041	0.952447
nomic-embed-text:latest	quran-embedding-prefect-nomic-embed-text-latest	768	137 Million	0.905570	0.905492

4.2 Performance across Proto-Languages

- A breakdown of the results for each of the embedding models is provided in the tables below:

Code

def per_class_scores(df, encoder, tar_col="target_idx", pred_col="knn_predictions"):
    """ per-class recall and precision computed over the whole of df.

    Scoring within a single language group would leave only one true class, so
    precision would collapse to 1/(number of distinct classes predicted).
    """
    labels = np.arange(len(encoder.classes_))
    names = encoder.classes_
    # '' is only in the encoder so heuristic labels can be transformed; it is
    # never a true or predicted class here.
    keep = names != ''
    rec = recall_score(df[tar_col], df[pred_col], labels=labels, average=None, zero_division=np.nan)
    prec = precision_score(df[tar_col], df[pred_col], labels=labels, average=None, zero_division=np.nan)
    index = pd.Index(names[keep], name="language_group")

    return pd.Series(rec[keep], index=index), pd.Series(prec[keep], index=index)

agg_recall = []
agg_precision = []

for c in collections:
    test = samples[c][samples[c]['Train/Test'].eq("Test")]
    rk, pk = per_class_scores(test, tar_encoder[c])

    agg_recall.append(rk.rename(f"UMAP+KNN - {c} - Recall"))
    agg_precision.append(pk.rename(f"UMAP+KNN - {c} - Precision"))

Code

agg_rec_res = pd.concat(agg_recall,axis=1)
agg_rec_desc = agg_rec_res.describe() 
agg_rec_desc.index = ['No. Proto-Languages','Avg. Recall', "Std. Recall", "Minimum Recall", "25th Percentile", "50th Percentile", "75th Percentile", "Maximum Recall" ]

agg_prec_res = pd.concat(agg_precision, axis=1)
agg_prec_desc = agg_prec_res.describe()
agg_prec_desc.index = ['No. Proto-Languages','Avg. Precision', "Std. Precision", "Minimum Precision", "25th Percentile", "50th Percentile", "75th Percentile", "Maximum Precision" ]

Recall per Approach

Code

style_dataframe_heatmap(agg_rec_res)

	UMAP+KNN - gemini-embedding-001 - Recall	UMAP+KNN - nomic-embed-text:latest - Recall	UMAP+KNN - nomic-embed-text-v2-moe:latest - Recall
language_group
Afroasiatic	0.99	0.92	0.97
Dravidian	1.00	0.94	0.70
Indo-European	0.97	0.99	0.97
Kra-Dai	0.97	0.70	0.44
Sino-Tibetan	0.99	0.99	0.79
Turkic	0.80	0.98	0.52
n/a	0.91	0.92	1.00

Precision per Approach

Code

style_dataframe_heatmap(agg_prec_res)

	UMAP+KNN - gemini-embedding-001 - Precision	UMAP+KNN - nomic-embed-text:latest - Precision	UMAP+KNN - nomic-embed-text-v2-moe:latest - Precision
language_group
Afroasiatic	0.99	0.97	0.96
Dravidian	1.00	0.96	0.89
Indo-European	0.96	0.99	0.84
Kra-Dai	0.99	0.81	0.96
Sino-Tibetan	0.98	1.00	0.94
Turkic	0.85	0.99	0.90
n/a	0.90	0.62	1.00

5.0 Lessons Learned

Multilingual support is not a predictor of performance: gemini-embedding-001 performed best, but nomic-embed-text:latest, which only supports English, came close, while nomic-embed-text-v2-moe:latest, which supports almost every language in the study, performed worst on recall and MI.
The weak spots are the small language groups: performance was weakest for language groups represented by only one or two languages: Kra-Dai (Thai) for nomic-embed-text:latest, and Turkic for nomic-embed-text-v2-moe:latest and gemini-embedding-001. Indo-European, which covers 11 of the 20 languages and roughly half of the sample, scored consistently high, which suggests that the majority group absorbs neighbouring points from smaller groups.